Imaging device and imaging system

ABSTRACT

An imaging device includes multiple pixels each having a photoelectric converter, a control unit that controls the pixels to output, from each pixel, a first signal based on charges generated by the photoelectric converter during first exposure time and a second signal based on charges generated by the photoelectric converter during second exposure time shorter than the first exposure time, a decision unit that decides whether or not a predetermined value is exceeded for each first signal output from the pixels and outputs a decision signal indicating a decision result, and a signal select unit that selects one of the first and second signals as an image forming signal of each pixel, and based on the decision signals of a target pixel and another pixel arranged in a predetermined region including the target pixel, the signal select unit selects the image forming signal of the target pixel.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an imaging device and an imagingsystem.

Description of the Related Art

In an imaging device used for an on-vehicle camera, a surveillancecamera, or the like, a scheme to generate an image having a wide dynamicrange by composing an image obtained by a relatively long exposure time(a long-time image) and an image obtained by a relatively short exposuretime (a short-time image) is used in general. Japanese PatentApplication Laid-Open No. 2006-191536 discloses a method that, when along-time image and a short-time image are composed, a compositionprocess is performed by referencing a saturation flag indicating whetheror not a signal of the long-time image is saturated and, when the signalof the long-time image is saturated, using a signal of the short-timeimage instead of the signal of the long-time image.

In the scheme disclosed in Japanese Patent Application Laid-Open No.2006-191536, however, image quality of an image obtained by composing along-time image and a short-time image may deteriorate due to an offsetoccurring at a joining part of the long-time image and the short-timeimage.

SUMMARY OF THE INVENTION

The present invention intends to provide an imaging device and animaging system that may acquire a high quality, high dynamic rangeimage.

According to one aspect of the present invention, provided is an imagingdevice including a plurality of pixels arranged over a plurality of rowsand a plurality of columns and each having a photoelectric converter, acontrol unit that controls the plurality of pixels so as to output, fromeach of the plurality of pixels, a first signal based on chargesgenerated by the photoelectric converter during a first exposure timeand a second signal based on charges generated by the photoelectricconverter during a second exposure time that is shorter than the firstexposure time, a decision unit that performs a decision process todecide whether or not a predetermined value is exceeded for the firstsignal output from each of the plurality of pixels and outputs adecision signal indicating a decision result, and a signal select unitthat selects one of the first signal and the second signal as an imageforming signal of each of the plurality of pixels, wherein based on thedecision signal of a target pixel and the decision signal of anotherpixel arranged in a predetermined region including the target pixel, thesignal select unit selects the image forming signal of the target pixel.

Further, according to another aspect of the present invention, providedis a signal processing device configured to process a signal output froman imaging element that includes a plurality of pixels each having aphotoelectric converter and outputs, from each of the plurality ofpixels, a first signal based on charges generated by the photoelectricconverter during a first exposure time and a second signal based oncharges generated by the photoelectric converter during a secondexposure time that is shorter than the first exposure time. The signalprocessing device includes a decision unit that decides whether or not apredetermined value is exceeded for the first signal output from each ofthe plurality of pixels and outputs a decision signal indicating adecision result, and a signal select unit that selects one of the firstsignal and the second signal as an image forming signal of each of theplurality of pixels, wherein based on the decision signal of a targetpixel and the decision signal of another pixel arranged in apredetermined region including the target pixel, the signal select unitselects the image forming signal of the target pixel.

Further, according to yet another aspect of the present invention,provided is an imaging system including an imaging device including animaging element that includes a plurality of pixels each having aphotoelectric converter and outputs, from each of the plurality ofpixels, a first signal based on charges generated by the photoelectricconverter during a first exposure time and a second signal based oncharges generated by the photoelectric converter during a secondexposure time that is shorter than the first exposure time, and a signalprocessing unit that processes a signal output from the imaging device.The signal processing unit includes a decision unit that decides whetheror not a predetermined value is exceeded for the first signal outputfrom each of the plurality of pixels and outputs a decision signalindicating a decision result, and a signal select unit that selects oneof the first signal and the second signal as an image forming signal ofeach of the plurality of pixels, wherein based on the decision signal ofa target pixel and the decision signal of another pixel arranged in apredetermined region including the target pixel, the signal select unitselects the image forming signal of the target pixel.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a general configuration of animaging device according to a first embodiment of the present invention.

FIG. 2 is a block diagram illustrating a general configuration of animaging element in the imaging device according to the first embodimentof the present invention.

FIG. 3 is a circuit diagram illustrating a configuration example ofpixels of the imaging element in the imaging device according to thefirst embodiment of the present invention.

FIG. 4 is a timing diagram illustrating a readout operation of theimaging element in the imaging device according to the first embodimentof the present invention.

FIG. 5 is a diagram illustrating an operation of a long-time signalsaturation decision unit in the imaging device according to the firstembodiment of the present invention.

FIG. 6A is a diagram illustrating a configuration of a signal selectunit in the imaging device according to the first embodiment of thepresent invention.

FIG. 6B and FIG. 6C are diagrams illustrating operations of the signalselect unit in the imaging device according to the first embodiment ofthe present invention.

FIG. 7A and FIG. 7B are diagrams illustrating operations of an HDRcomposition unit in the imaging device according to the first embodimentof the present invention.

FIG. 8A and FIG. 8B are diagrams illustrating advantages of the imagingdevice according to the first embodiment of the present invention.

FIG. 9A, FIG. 9B, FIG. 10A, and FIG. 10B are diagrams illustratinggeneration methods of long/short select signal in an imaging deviceaccording to a second embodiment of the present invention.

FIG. 11 is a block diagram illustrating a general configuration of animaging element in an imaging device according to a third embodiment ofthe present invention.

FIG. 12 is a circuit diagram illustrating a configuration example ofpixels of the imaging element in the imaging device according to thethird embodiment of the present invention.

FIG. 13 is a timing diagram illustrating a readout operation of theimaging element in the imaging device according to the third embodimentof the present invention.

FIG. 14 is a timing diagram illustrating a readout operation on one rowof the imaging element in the imaging device according to the thirdembodiment of the present invention.

FIG. 15 is a diagram illustrating a configuration and an operation of asignal processing unit in the imaging device according to the thirdembodiment of the present invention.

FIG. 16 is a block diagram illustrating a general configuration of animaging element in an imaging device according to a fourth embodiment ofthe present invention.

FIG. 17 is a timing diagram illustrating a readout operation on one rowof the imaging element in the imaging device according to the fourthembodiment of the present invention.

FIG. 18A and FIG. 18B are circuit diagrams illustrating configurationexamples of the signal processing unit of the imaging element in theimaging device according to the fourth embodiment of the presentinvention.

FIG. 19 is a block diagram illustrating a general configuration of animaging system according to a fifth embodiment of the present invention.

FIG. 20A is a diagram illustrating a configuration example of an imagingsystem according to a sixth embodiment of the present invention.

FIG. 20B is a diagram illustrating a configuration example of a movableobject according to the sixth embodiment of the present invention.

DESCRIPTION OF THE EMBODIMENTS First Embodiment

The configuration of an imaging device according to a first embodimentof the present invention will be described with reference to FIG. 1 toFIG. 3. FIG. 1 is a block diagram illustrating a general configurationof the imaging device according to the present embodiment. FIG. 2 is ablock diagram illustrating a general configuration of an imaging elementin the imaging device according to the present embodiment. FIG. 3 is acircuit diagram illustrating a configuration example of pixels of theimaging element in the imaging device according to the presentembodiment.

As illustrated in FIG. 1, an imaging device 300 according to the presentembodiment includes an imaging element 100 and a signal processing unit200.

The imaging element 100 converts an incident light signal (an objectimage) received via an optical system (not illustrated) into an electricsignal and outputs the electric signal. The imaging element 100 may beformed of a so-called single-plate color sensor in which color filters(hereafter, also referred to as “CF”) are arranged on a CMOS imagesensor or a CCD image sensor, for example. The imaging element 100 isnot necessarily required to be a color sensor and may be a monochromesensor.

The imaging element 100 outputs a signal based on charges generatedduring a first exposure time and a signal based on charges generatedduring a second exposure time. The first exposure time is relativelylonger than the second exposure time. The second exposure time isrelatively shorter than the first exposure time. In the descriptionbelow, the signal based on charges generated during the first exposuretime may be referred to as “long-time signal”, and the signal based oncharges generated during the second exposure time may be referred to as“short-time signal”. The long-time signal and the short-time signal aresignals used for obtaining an image signal having a wide dynamic range.

The signal processing unit 200 includes a pre-processing unit 210, along-time signal saturation decision unit 220, a long-timesignal/decision signal holding unit 230, a short-time signal holdingunit 240, a signal select unit 250, and an HDR composition unit 260.

The pre-processing unit 210 performs a pre-process of signal processingon an output signal from the imaging element 100. When the output signalfrom the imaging element 100 is an analog signal, an analog-to-digital(A/D) conversion process on the output signal from the imaging element100 may be performed in the pre-processing unit 210. In the presentembodiment, 12-bit A/D conversion is performed on the output signal fromthe imaging element 100. The pre-processing unit 210 suitably performscorrection (pre-processing) such as offset (OFFSET) correction, gain(GAIN) correction, or the like on the output signal (input signal Din)of the imaging element 100 and generates the corrected output signal(data Dout). This process is typically expressed by Equation (1) below.

Dout=(Din−OFFSET)×GAIN  (1)

Correction in the pre-processing unit 210 can be performed in variousunits. For example, a case where correction is performed on a pixel 12basis, a case where correction is performed on a column amplifier basis,a case where correction is performed on an analog-to-digital conversion(ADC) unit basis, a case where correction is performed on an outputamplifier basis, or the like may be considered. By performing correctionof the output signal from the imaging element 100, it is possible toreduce so-called fixed pattern noise and obtain a higher quality image.

The pre-processing unit 210 performs pre-processing on each of along-time signal and a short-time signal output from the imaging element100, separates the processed long-time signal and the processedshort-time signal from each other, and transmits the separated signalsto a post-processing unit. Specifically, the pre-processing unit 210transmits the processed long-time signal to the long-time signalsaturation decision unit 220 and transmits the processed short-timesignal to the short-time signal holding unit 240.

The long-time signal saturation decision unit 220 performs a decisionprocess to decide whether or not the long-time signal is saturated andtransmits a saturation decision signal indicating a decision resulttogether with the long-time signal to the post-processing unit.

The long-time signal/decision signal holding unit 230 holds a long-timesignal and a saturation decision signal transmitted from the long-timesignal saturation decision unit 220. Further, the short-time signalholding unit 240 holds a short-time signal transmitted from thepre-processing unit 210. A timing when a long-time signal is transmittedfrom a pixel 12 is different from a timing when a short-time signal istransmitted from the same pixel 12. The long-time signal/decision signalholding unit 230 is a memory that temporarily holds a long-time signaland a decision signal in order to output a long-time signal from a pixeland a short-time signal from the same pixel to the post-processing unitat the same time. Similarly, a short-time signal holding unit 240 is amemory that temporarily holds a short-time signal in order to output along-time signal from a pixel and a short-time signal from the samepixel to the post-processing unit at the same time. The long-timesignal/decision signal holding unit 230 and the short-time signalholding unit 240 include line memories formed of SRAM for 10 rows, forexample.

The signal select unit 250 selects one of the long-time signaltransmitted from the long-time signal saturation decision unit 220 andthe short-time signal transmitted from the short-time signal holdingunit 240 based on the saturation decision signal held in the long-timesignal/decision signal holding unit 230 and outputs the selected signalto the HDR composition unit 260. The selected signal is used as a signalfor image forming in the pixel.

The HDR composition unit 260 performs a high dynamic range process tocompose a high dynamic range image by using a select flag transmittedfrom the signal select unit 250 and the selected signal and outputs HDRcompression data.

As illustrated in FIG. 2, the imaging element 100 includes a pixel unit10, a vertical scanning circuit 20, a readout circuit unit 30, a memoryunit 40, a counter 46, a horizontal scanning circuit 50, a signalprocessing unit 60, and a timing generation circuit 70.

In the pixel unit 10, the plurality of pixels 12 are arranged in amatrix over a plurality of rows and a plurality of columns. In the pixelunit 10, for example, 1920 pixels in the column direction by 1080 pixelsin the row direction, namely, 2073600 pixels in total are arranged. Thenumber of pixels arranged in the pixel unit 10 is not limited, and alarger number of pixels or a smaller number of pixels may be arranged.

On each row of the pixel unit 10, a control line 14 is arrangedextending in a first direction (the horizontal direction in FIG. 2). Thecontrol line 14 is connected to the pixel 12 aligned in the firstdirection, respectively, to form a signal line common to these pixels12. The first direction in which the control line 14 extends may bereferred to as a row direction or a horizontal direction.

On each column of the pixel unit 10, a vertical signal line 16 isarranged extending in a second direction intersecting the firstdirection (the vertical direction in FIG. 2). The vertical signal line16 is connected to the pixels 12 aligned in the second direction,respectively, to form a signal line common to these pixels 12. Thesecond direction in which the vertical signal line 16 extends may bereferred to as a column direction or a vertical direction.

In FIG. 2, a case where the pixel unit 10 includes the pixels 12 on ncolumns from the first column to the n-th column is considered, and acolumn number is attached to the reference of the vertical signal line16 on each column. For example, the vertical signal line 16 on the firstcolumn is labeled with a reference “16-1”, and the vertical signal line16 on the n-th column is labeled with a reference “16-n”. In thedescription below, the same notation is used for elements provided inassociation with columns of the pixel unit 10.

The control line 14 on each row is connected to the vertical scanningcircuit 20. The vertical scanning circuit 20 is a control unit thatsupplies, to the pixel 12 via the control line 14, a control signal usedfor driving a readout circuit in the pixel 12 on a row basis when thepixel signal is read out. Pixel signals of the pixel 12 belonging to arow selected by a control signal supplied from the vertical scanningcircuit 20 (selected row) are output to the vertical signal lines 16 oncolumns corresponding to these pixels 12 at the same time.

The vertical signal line 16 on each column is connected to the readoutcircuit unit 30. The readout circuit unit 30 includes an amplifier unit32, a comparison unit 34, and a reference signal generation circuit 36.The amplifier unit 32 includes n amplifiers 33-1 to 33-n provided inassociation with respective columns of the pixel unit 10. The amplifiers33-1 to 33-n amplify pixel signals output from the pixels 12 via thevertical output lines 16-1 to 16-n corresponding to respective columns.The comparison unit 34 includes n comparators 35-1 to 35-n provided inassociation with respective columns of the pixel unit 10. Thecomparators 35-1 to 35-n compare the levels of pixel signals output fromthe amplifiers 33-1 to 33-n corresponding to respective columns with thelevel of a reference signal output from the reference signal generationcircuit 36 and outputs a comparison result.

The memory unit 40 includes n memories 42-1 to 42-n provided inassociation with respective columns of the pixel unit 10. The memories42-1 to 42-n receive signals output from the comparators 35-1 to 35-n oncolumns corresponding to respective memories and latch and hold a countsignal output from the counter 46 in response to the received signal.The reference signal output from the reference signal generation circuit36 is a signal whose signal level changes with time at a constant rate.The count signals held in the memories 42-1 to 42-n correspond todigital pixel signals obtained by performing analog-to-digital (AD)conversion on analog pixel signals output from the pixel 12.

The horizontal scanning circuit 50 is a control unit that supplies, tothe memory unit 40, a control signal used for sequentially selecting thememories 42-1 to 42-n on respective columns of the memory unit 40. Thememories 42-1 to 42-n that have received control signals from thehorizontal scanning circuit 50 transfer the held digital pixel signalsto the signal processing unit 60. The signal processing unit 60 performspredetermined digital signal processing on the digital pixel signal ofeach column transferred from the memory unit 40 and outputs theprocessed signal to the outside of the imaging element 100.

The timing generation circuit 70 is a circuit unit for supplying, to thevertical scanning circuit 20, the readout circuit unit 30, the memoryunit 40, the horizontal scanning circuit 50, and the like, controlsignals that control the operations or timings thereof. Some or all ofthe control signals supplied to the vertical scanning circuit 20, thereadout circuit unit 30, memory unit 40, the horizontal scanning circuit50, and the like may be supplied from the outside of the imaging element100.

As illustrated in FIG. 3, each of the pixels 12 includes a photodiodePD, a transfer transistor MTX, a reset transistor MRS, an amplifiertransistor MSF, and a select transistor MSEL. The photodiode PD has theanode connected to a reference voltage node and the cathode connected tothe source of the transfer transistor MTX. The drain of the transfertransistor MTX is connected to the source of the reset transistor MRSand the gate of the amplifier transistor MSF. The connection node of thedrain of the transfer transistor MTX, the source of the reset transistorMRS, and the gate of the amplifier transistor MSF is a so-calledfloating diffusion FD. The floating diffusion FD includes a capacitancecomponent, functions as a charge holding portion, and forms acharge-to-voltage conversion unit formed of the capacitance component.The drain of the reset transistor MRS and the drain of the amplifiertransistor MSF are connected to a power supply voltage node (voltageVDD). The source of the amplifier transistor MSF is connected to thedrain of the select transistor MSEL. The source of the select transistorMSEL that is also the output node of the pixel 12 is connected to thevertical signal line 16.

The photodiode PD is a photoelectric converter that generates chargescorresponding to the light amount of an incident light. When an opticalimage of an object enters the pixel unit 10, the photodiode PD of eachpixel 12 converts (photoelectrically converts) the incident light intoan amount of charges in accordance with the light amount and accumulatesthe generated charges. When turned on, the transfer transistor MTXtransfers charges held by the photodiode PD to the floating diffusionFD. The floating diffusion FD has a voltage corresponding to the amountof charges transferred from the photodiode PD by charge-to-voltageconversion caused by the capacitance component. The amplifier transistorMSF is configured such that the voltage VDD is supplied to the drain anda bias current is supplied to the source from a current source (notillustrated) via the select transistor MSEL, and forms an amplifier unit(source follower circuit) whose gate is the input node. Thus, theamplifier transistor MSF outputs a signal based on the voltage of thefloating diffusion FD to the vertical signal line 16 via the selecttransistor MSEL. When turned on, the reset transistor MRS resets thefloating diffusion FD to a voltage corresponding to the voltage VDD.

In the case of the pixel 12 of the circuit configuration as illustratedFIG. 3, the control line 14 on each row includes a signal line connectedto the gate of the transfer transistor MTX, a signal line connected tothe gate of the reset transistor MRS, and a signal line connected to thegate of the select transistor MSEL. A control signal ϕTX is suppliedfrom the vertical scanning circuit 20 to the transfer transistor MTX viathe control line 14. A control signal ϕRES is supplied from the verticalscanning circuit 20 to the reset transistor MRS via the control line 14.A control signal ϕSEL is supplied from the vertical scanning circuit 20to the select transistor MSEL via the control line 14. A plurality ofpixels 12 in the pixel unit 10 are controlled by the control signalsϕTX, ϕRES, and ϕSEL supplied from the vertical scanning circuit 20 on arow basis. When each transistor of the pixel 12 is formed of ann-channel transistor, the corresponding transistor is in an on-statewhen the above control signal is at a High level (H level), and thecorresponding transistor is in an off-state when the above controlsignal is at a Low level (L level).

Next, the operation timing of the imaging element 100 will be describedby using FIG. 4. FIG. 4 is a timing diagram illustrating a readoutoperation of the imaging element in the imaging device according to thepresent embodiment.

As illustrated in FIG. 4, the imaging element 100 performs lineinterleave drive to perform a readout operation of a long-time signaland a readout operation of a short-time signal in an alternating manneron a row basis during one frame.

A readout operation of a signal in a particular frame is started fromtime T40, for example. Time T40 is the time when a readout operation ofa long-time signal of the pixel 12 on the first row in the frame isstarted. Readout operations of the long-time signal of the pixel 12 ofthe second and subsequent rows are performed sequentially on a row basisat a predetermined interval.

A readout operation of a short-time signal of the pixel 12 on the firstrow is started, for example, from time T41 after the end of the readoutoperation of the long-time signal of the pixel 12 on the third row. Areadout operation of a long-time signal of the pixel 12 on the fourthrow is started from time T42 after the end of the readout operation ofthe short-time signal of the pixel 12 on the first row.

In such a way, on and after time T41, the readout operations of thelong-time signal and the short-time signal are performed in analternating manner on a row basis in the order of readout of theshort-time signal of the pixels 12 on the first row, readout of thelong-time signal of the pixel 12 on the fourth row, readout of theshort-time signal on the second row, and readout of the long-time signalon the fifth row. Upon the completion of the readout operation of theshort-time signal of the pixel 12 on the m-th row, which is the lastrow, the readout operation of the frame is completed.

At time T43 after the end of the readout operation of the short-timesignal of the pixel 12 on the m-th row, a readout operation of the nextframe is started. The operation on and after time T43 is the same as thereadout operation in the previous frame, that is, in the period fromtime T40 to T43. The period from time T43 to T44 is a period in which areadout operation of the long-time signal of the pixel 12 on the firstrow is performed. The length of the period from time T42 to T44 is theexposure time (accumulation time) T1 for generating signal charges as along-time signal in the pixel 12 on the first row. The period from timeT45 to T46 is a period in which a readout operation of a short-timesignal of the pixel 12 on the first row is performed. The length of theperiod from time T44 to T46 is the exposure time (accumulation time) T2for generating signal charges as a short-time signal in the pixel 12 onthe first row.

In such a way, the long-time signal and the short-time signal read outfrom the imaging element 100 are input to the pre-processing unit 210 ofthe signal processing unit 200. The pre-processing unit 210 performspredetermined pre-processing for each of the long-time signal and theshort-time signal, then transmits the processed long-time signal to thelong-time signal saturation decision unit 220, and transmits theprocessed short-time signal to the short-time signal holding unit 240.

Next, the operation of the long-time signal saturation decision unit 220will be described by using FIG. 5. FIG. 5 is a diagram illustrating theoperation of the long-time signal saturation decision unit 220 in theimaging device according to the present embodiment.

As illustrated in FIG. 5, the signal level of the long-time signal andthe short-time signal increases as the amount of incident light on thepixel 12 increases. When the amount of signal charges generated in thephotodiode PD exceeds the upper limit of the charge amount that can beaccumulated in the photodiode PD (saturation charge amount), the signallevel is saturated. A long-time signal whose accumulation time of signalcharges is relatively longer than a short-time signal has a largersignal level than the short-time signal at the same light amount andthus is saturated at less light amount than the short-time signal.

In the long-time signal saturation decision unit 220, a saturationthreshold value that is a reference for deciding whether or not theoutput signal is saturated is set in advance. As illustrated in FIG. 5,for example, the saturation threshold value may be set to a signal levelslightly lower than the output signal level corresponding to thesaturation charge amount of the photodiode PD.

The long-time signal saturation decision unit 220 compares data of along-time signal received from the pre-processing unit 210 with a presetsaturation threshold value. As a result, when a data value of along-time signal exceeds the saturation threshold value, a saturationdecision signal JL is set to 1, and when a data value of a long-timesignal is less than or equal to the threshold value, the saturationdecision signal JL is set to 0. The long-time signal saturation decisionunit 220 then transmits a long-time signal and the saturation decisionsignal JL indicating the decision result to the long-timesignal/decision signal holding unit 230. The long-time signal/decisionsignal holding unit 230 holds a long-time signal and a saturationdecision signal received from the long-time signal saturation decisionunit 220.

Next, the operation of the signal select unit 250 will be described byusing FIG. 6A to FIG. 6C. FIG. 6A is a diagram illustrating aconfiguration example of the signal select unit 250 in the imagingdevice according to the present embodiment. FIG. 6B and FIG. 6C arediagrams illustrating the operation of the signal select unit 250 in theimaging device according to the present embodiment.

As illustrated in FIG. 6A, the signal select unit 250 includes along/short select signal generation unit 252 and a selector 254. Thelong/short select signal generation unit 252 holds the saturationdecision signal JL received from the long-time signal/decision signalholding unit 230 for a plurality of pixels. In the case of a monochromesensor, the long/short select signal generation unit 252 holds thesaturation decision signal JL corresponding to the pixels 12 in apredetermined region including a plurality of columns adjacent to eachother. For example, the long/short select signal generation unit 252holds a saturation decision signal JL (N−1) corresponding to the pixel12 on the (N−1)-th column, a saturation decision signal JL (N)corresponding to the pixel 12 on the N-th column, and a saturationdecision signal JL (N+1) corresponding to the pixel 12 on the (N+1)-thcolumn. Then, the long/short select signal generation unit 252 generatesa long/short select signal JJ (N) corresponding to the pixel 12 on theN-th column based on the saturation decision signals JL (N−1), JL (N),and JL (N+1) and transmits the long/short select signals to the selector254.

FIG. 6B is an example of a truth table illustrating the relationshipbetween the saturation decision signals JL (N−1), JL (N), and JL (N+1)as input signals and the long/short select signal JJ (N) as an outputsignal. In the truth table, ϕ denotes “don't care”.

The long/short select signal generation unit 252 outputs 1 as along/short select signal JJ (N) unconditionally if the saturationdecision signal JL (N) is 1. Even if the saturation decision signal JL(N) is 0, if one of the saturation decision signal JL (N−1) and thesaturation decision signal JL (N+1) is 1, the long/short select signalgeneration unit 252 outputs 1 as the long/short select signal JJ (N).Only if all the saturation decision signals JL (N−1), JL (N), and JL(N+1) are 0, the long/short select signal generation unit 252 outputs 0as the long/short select signal JJ (N).

The selector 254 selects and outputs one of the long-time signal (N)received from the long-time signal/decision signal holding unit 230 andthe short-time signal (N) received from the short-time signal holdingunit 240 in accordance with the long/short select signal JJ (N) receivedfrom the long/short select signal generation unit 252. For example, ifthe long/short select signal JJ (N) is 0, the selector 254 outputs thelong-time signal (N), and if the long/short select signal JJ (N) is 1,the selector 254 outputs the short-time signal (N).

That is, the signal select unit 250 selects the short-time signal (N) ofthe pixel 12 on the N-th column when the saturation decision has beenmade on the long-time signal (N) of the pixel 12 on the N-th column.Even when no saturation decision has been made on the long-time signal(N) of the pixel 12 on the N-th column, when saturation decision hasbeen made on the long-time signal (N−1) or the long time-signal (N+1) ofthe pixel 12 on the adjacent columns, the signal select unit 250 selectsthe short-time signal. The select signal unit 250 selects the long-timesignal (N) of the pixel 12 on the N-th column only when no saturationdecision has been made on the long-time signal (N) of the pixel 12 onthe N-th column and the long-time signal (N−1) and the long-time signal(N+1) of the pixels 12 on columns adjacent to the N-th column.

FIG. 6C illustrates an example of the operation in the signal selectunit 250. FIG. 6C illustrates an example of values of the long-timesignal and the short-time signal output from the pixels 12 on the(N−3)-th to the (N+3)-th columns, the saturation decision signal JLcorresponding to these long-time signals, and the long/short selectsignal JJ generated by the long/short select signal generation unit 252.In the example of FIG. 6C, the saturation decision signals JL (N−1), JL(N), and JL (N+1) are 1, 0, and 1, respectively, indicated in the framesurrounded by a dotted line. In this case, the long/short select signalJJ (N) is 1 as indicated in the truth table of FIG. 6B.

In such a way, the signal select unit 250 outputs the long/short selectsignal JJ (N) generated by the long/short select signal generation unit252 and one of the long-time signal (N) and the short-time signal (N)selected by the selector 254 to the HDR composition unit 260.

Next, the operation of the HDR composition unit 260 will be described byusing FIG. 7A and FIG. 7B. FIG. 7A and FIG. 7B are diagrams illustratingoperations of the HDR composition unit 260 in the imaging deviceaccording to the present embodiment.

The HDR composition unit 260 first performs an HDR composition process.FIG. 7A is a schematic diagram illustrating the outline of the HDRcomposition process performed in the HDR composition unit 260. The HDRcomposition process generates data in accordance with the long/shortselect signal JJ on a pixel 12 basis. That is, if the long/short selectsignal JJ is 0, a value of a long-time signal D1 is directly used asdata of the corresponding pixel 12. If the long/short select signal JJis 1, a value obtained by multiplying a short-time signal D2 by a rateT1/T2 is used as data of the corresponding pixel 12. Here, T1 is theaccumulation time of signal charges from which the long-time signal isgenerated, and T2 is the accumulation time of signal charges from whichthe short-time signal is generated.

In the HDR composition unit 260, an HDR compression process is performedafter the HDR composition process. FIG. 7B is a schematic diagramillustrating the outline of the HDR compression process performed in theHDR composition unit 260. In the HDR compression process, data generatedby the HDR composition process (for example, 20 bits) is compressed into12-bit data (=4096 [LSB]), for example. FIG. 7B illustrates an examplein which data generated by the HDR composition process is compressedinto 12-bit data by four times of polyline compression.

Next, the advantage of the imaging device according to the presentembodiment will be described by using FIG. 8A and FIG. 8B. FIG. 8A andFIG. 8B are diagrams illustrating the advantage of the imaging deviceaccording to the present embodiment.

The value of the long-time signal D1 and the value obtained bymultiplying the short-time signal D2 by the ratio T1/T2 are ideally thesame. However, these values are not always the same due to a shift ofthe linearity, a shift of a black level of the circuit, or the like inpractice.

In such a case, for example, as illustrated in FIG. 7A, a leveldifference will occur at the joining part between the line indicated byD1 and the line indicated by D2×T1/T2. With a level difference occurringat the joining part of the lines, variation between the value based onthe long-time signal D1 and the value based on the short-time signal D2will be notable near the saturation threshold value.

FIG. 8A is a graph illustrating a result obtained by performing an HDRcomposition process on signals of the pixels 12 from the (N−3)-th columnto the (N+3)-the column of FIG. 6C by using the saturation decisionsignal JL. In this example, a value based on the short-time signal D2and a value based on the long-time signal D1 occur in an alternatingmanner as the values of the pixels 12 from the (N−1)-th column to the(N+3)-the column. In such a case, with a level difference as illustratedin FIG. 7A occurring at the joining part of the lines, the variation invalues between the pixels 12 adjacent to each other will be emphasized.

FIG. 8B is a graph illustrating a result obtained by performing an HDRcomposition process on signals of the pixels 12 from the (N−3)-th columnto the (N+3)-the column of FIG. 6C by using the long/short select signalJJ. In the present embodiment, even with a signal of the pixel 12 inwhich the long-time signal D1 is not saturated, when the long-timesignal D1 of the adjacent pixels 12 is saturated, a pixel value iscalculated by using the value of the short-time signal D2. Therefore,all the values of the pixels 12 from the (N−1)-th column to the(N+3)-the column will be values based on the short-time signal D2.Therefore, in the imaging device according to the present embodiment, anHDR composition signal having small spatial variation can be obtained asillustrated in FIG. 8B.

As described above, according to the present embodiment, when a highdynamic range image is composed by using a long-time image and ashort-time image, image quality at a joining part between the long-timeimage and the short-time image can be improved.

Second Embodiment

An imaging device according to a second embodiment of the presentinvention will be described with reference to FIG. 9A to FIG. 10B. Thesame component as that of the imaging device according to the firstembodiment is labeled with the same reference, and the descriptionthereof will be omitted or simplified. FIG. 9A, FIG. 9B, FIG. 10A, andFIG. 10B are diagrams illustrating a generation method of a long/shortselect signal in the imaging device according to the present embodiment.

The imaging device according to the present embodiment is the same asthe imaging device according to the first embodiment except that thepixel 12 that provides the saturation decision signal JL used ingenerating the long/short select signal JJ is different. That is, in thefirst embodiment, the example in which the HDR composition process isperformed taking into consideration of the long/short select signal JJgenerated based on the saturation decision signals JL of adjacent twopixels 12 arranged on the same row as the target pixel 12 has beenillustrated. In contrast, in the present embodiment, an HDR compositionprocess is performed taking into consideration of the long/short selectsignal JJ generated based on the saturation decision signals JL of otherpixels 12 included in a predetermined region including the target pixel12. The predetermined region includes a plurality of columns adjacent toeach other and a plurality of rows adjacent to each other.

FIG. 9A and FIG. 9B illustrate an application example to a monochromesensor. FIG. 9A illustrates pixel arrangement in a monochrome sensor. Inthe pixel unit 10 of the monochrome sensor, pixels having sensitivityover a visible wavelength range (denoted as W in FIG. 9A) are arrangedin a matrix. FIG. 9A illustrates 64 pixels 12 arranged from the (M−4)-throw to the (M+3)-th row and the (N−3)-th column to the (N+4)-th columnout of the plurality of pixels 12 forming the pixel unit 10. Note thatit can be considered that all the pixels 12 of the monochrome sensorhave the color filters of the same color.

A pixel arranged at the M-th row, N-th column (a pixel surrounded by abold line in FIG. 9A) is now focused on, and eight pixels 12 neighborthe circumference of the pixel. By generating the long/short selectsignal JJ based on the saturation decision signals JL provided fromthese eight pixels 12, it is possible to obtain an HDR composite signalhaving smaller spatial variation.

FIG. 9B illustrates one example of the process in the signal select unit250 in the case of a monochrome sensor. In the example of FIG. 9B, aregion of three rows by three columns including a pixel of interest atthe center is extracted, and when there is at least one pixel whosesaturation decision signal JL is 1 out of the eight pixels around thepixel of interest, the long/short select signal JJ is set to 1.

FIG. 10A and FIG. 10B illustrate an application example to a colorsensor. FIG. 10A illustrates the pixel arrangement in a color sensor. Inthe pixel unit 10 of the color sensor, for example, R pixels having highsensitivity to red, G pixels having high sensitivity to green, and Bpixels having high sensitivity to blue are arranged in accordance withso-called RGB Bayer color filter arrangement as illustrated in FIG. 10A.FIG. 10A illustrates 64 pixels 12 arranged from the (M−4)-th row to the(M+3)-th row and the (N−4)-th column to the (N+3)-th column out of theplurality of pixels 12 forming the pixel unit 10.

FIG. 10B illustrates one example of the process in the signal selectunit 250 in the case of a color sensor. Also in the example of FIG. 10B,a region of three rows by three columns including a pixel of interest atthe center is extracted. In the color sensor, however, since processesare performed on a pixel basis having a color filter of the same colorin a color sensor, pixels from which saturation decision signals JL areacquired are selected from pixels having color filters of the same coloras the pixel of interest. For example, when the R pixel is focused on, Rpixels are arranged on the (N−4)-th column, the (N−2)-th column, theN-th column, and the (N+2)-th column on the (M−4)-th row, the (M−2)-throw, the M-th row, and the (M+2)-th row. Accordingly, for example, whenthe R pixel (the pixel surrounded by a bold line in FIG. 9A) arranged atthe M-th row, the N-th column is a pixel of interest, the (M−2)-th rowand (M+2)-th row are selected as rows around the R pixel of interest,and the (N−2)-th column and (N+2)-th column are selected as columnsaround the R pixel of interest.

In the region of three rows by three columns selected in such a way,eight R pixels are arranged around each pixel of interest at the center.Therefore, by generating the long/short select signal JJ based on thesaturation decision signal JL provided from these eight R pixels, it ispossible to obtain an HDR composite signal having smaller spatialvariation. In the example of FIG. 10B, when there is at least one pixelwhose saturation decision signal JL is 1 out of the eight pixels aroundthe pixel of interest, the long/short select signal JJ is set to 1.

While the case of the R pixel is described here, the same process canapply to the G pixel and the B pixel.

Note that, while the example in which a region of three rows by threecolumns including a pixel of interest at the center is extracted and thelong/short select signal JJ is acquired based on the saturation decisionsignal JL of other pixels included in this region has been illustratedin the present embodiment, a region from which the saturation decisionsignal JL is acquired is not limited thereto. For example, a region offive rows by five columns may be extracted, and the saturation decisionsignal JL may be acquired from more pixels.

Further, while the long/short select signal JJ is set to 1 when there isat least one pixel whose saturation decision signal JL is 1 out of theeight pixels around the pixel of interest in the present embodiment, acriterion for generating the long/short select signal JJ is also notlimited thereto. For example, the long/short select signal JJ may be setto 1 when the number of pixels having the saturation decision signal JLof 1 is larger than the number of pixels having the saturation decisionsignal JL of 0.

As described above, according to the present embodiment, when composinga high dynamic range image by using a long-time image and a short-timeimage, image quality at a joining part between the long-time image andthe short-time image can be improved.

Third Embodiment

An imaging device according to a third embodiment of the presentinvention will be described with reference to FIG. 11 to FIG. 15. Thesame component as that of the imaging device according to the first andsecond embodiments is labeled with the same reference, and thedescription thereof will be omitted or simplified.

First, the structure of the imaging device according to the presentembodiment will be described by using FIG. 11 and FIG. 12. FIG. 11 is ablock diagram illustrating a general configuration of an imaging elementin the imaging device in accordance with the present embodiment. FIG. 12is a circuit diagram illustrating a configuration example of pixels ofthe imaging element in the imaging device in accordance with the presentembodiment. Note that the entire configuration of the imaging deviceaccording to the present embodiment is basically the same as the imagingdevice according to the first embodiment illustrated in FIG. 1.

As illustrated in FIG. 11, the imaging element 100 of the imaging deviceaccording to the present embodiment is basically the same as the imagingelement 100 of the imaging device according to the first embodimentillustrated in FIG. 2 except that long-time signal saturation decisionunits 38 are further provided in association with respective columns ofthe pixel unit 10. That is, the imaging device according to the presentembodiment is different from the imaging device according to the firstembodiment, which performs saturation decision of a long-time signal atthe signal processing unit 200, in that the saturation decision of along-time signal is performed at the imaging element 100 in parallelwith respect to columns.

The memory 42 on each column includes a JL-memory 44-1, an N-memory44-2, an S-memory 44-3, an N-memory 44-4, and an S-memory 44-5. TheJL-memory 44-1 is a memory for holding the saturation decision signal JLoutput from the long-time signal saturation decision unit 38. TheN-memory 44-2 is a memory for holding, as a digital signal N1, anN-signal of a long-time signal output from the pixel 12. The S-memory44-3 is a memory for holding, as a digital signal S1, an S-signal of along-time signal output from the pixel 12. The N-memory 44-4 is a memoryfor holding, as a digital signal N2, an N-signal of a short-time signaloutput from the pixel 12. The S-memory 44-5 is a memory for holding, asa digital signal S2, an S-signal of a short-time signal output from thepixel 12.

A signal Vvl of each of the pixels 12 is read out to the vertical signalline 16 on the corresponding column on a row basis by the verticalscanning circuit 20. The amplifier 33 on each column amplifies thesignal Vvl read out from the pixel 12 via the vertical signal line 16 onthe corresponding column at a predetermined gain and outputs theamplified signal Vamp to the comparator 35 and the long-time signalsaturation decision unit 38. The long-time signal saturation decisionunit 38 on each column decides whether or not the level of the long-timesignal amplified by the amplifier 33 on the corresponding column exceedsthe level of a preset predetermined saturation threshold value andoutputs the saturation decision signal JL indicating a decision result.The saturation decision signal JL is held in the JL-memory 44-1 on thememory 42 on the corresponding column.

The reference signal generation circuit 36 outputs, to the comparator35, a reference signal Vr whose signal level changes at a constant ratewith time. The comparator 35 on each column compares the signal level ofsignal Vamp output from the amplifier 33 on the corresponding columnwith the signal level of the reference signal Vr and outputs the signalVcmp indicating a comparison result. The counter 46 outputs a countsignal indicating a count value that is a counting value of the clocksignal to the memory 42 on each column. The memory 42 on each columnholds, as a digital signal obtained by performing AD conversion on thesignal Vcmp, a count signal received from the counter 46 when the signallevel of the signal Vcmp changes. The memory 42 holds the digital signalobtained by AD conversion in any of the N-memory 44-2, the S-memory44-3, the N-memory 44-4, and the S-memory 44-5 in accordance with thetype of the signal.

As illustrated in FIG. 12, each of the pixels 12 includes a photodiodePD, transfer transistors MGS1, MGS2, MTX1, and MTX2, and a charge draintransistor MOFD. Each of the pixels 12 further includes a resettransistor MRS, an amplifier transistor MSF, and a select transistorMSEL.

The photodiode PD has the anode connected to a reference voltage nodeand the cathode connected to the source of the transfer transistor MGS1,the source of the transfer transistor MGS2, and the source of the chargedrain transistor MOFD. The drain of the transfer transistor MGS1 isconnected to the source of the transfer transistor MTX1. The connectionnode of the drain of the transfer transistor MGS1 and the source of thetransfer transistor MTX1 includes a capacitance component and functionsas a charge holding portion MEM1. The drain of the transfer transistorMGS2 is connected to the source of the transfer transistor MTX2. Theconnection node of the drain of the transfer transistor MGS2 and thesource of the transfer transistor MTX2 includes a capacitance componentand functions as a charge holding portion MEM2.

The drain of the transfer transistor MTX1 and the drain of the transfertransistor MTX2 are connected to the source of the reset transistor MRSand the gate of the amplifier transistor MSF. The connection node of thedrain of the transfer transistor MTX1, the drain of the transfertransistor MTX2, the source of the reset transistor MRS, and the gate ofthe amplifier transistor MSF is a so-called floating diffusion FD. Thefloating diffusion FD includes a capacitance component, functions as acharge holding portion, and forms a charge-to-voltage conversion unitmade of such a capacitance component. The drain of the charge draintransistor MOFD, the drain of the reset transistor MRS, and the drain ofthe amplifier transistor MSF are connected to a power supply node(voltage VDD). The source of the amplifier transistor MSF is connectedto the drain of the select transistor MSEL. The source of the selecttransistor MSEL, which is also the output node of the pixel 12, isconnected to the vertical signal line 16.

Once an optical image of an object enters the pixel unit 10, thephotodiode PD of each pixel 12 converts (photoelectrically converts) theincident light into an amount of charges in accordance with the lightamount thereof and accumulates the generated charges. When turned on,the charge drain transistor MOFD drains charges of the photodiode PD toreset the photodiode PD to a voltage in accordance with the voltage VDD.By turning off the charge drain transistor MOFD, it is possible to startaccumulation of charges in the photodiode PD. When turned on, thetransfer transistor MGS1 transfers charges of the photodiode PD to thecharge holding portion MEM1. When turned on, the transfer transistorMTX1 transfers charges of the charge holding portion MEM1 to thefloating diffusion FD. Similarly, when turned on, the transfertransistor MGS2 transfers charges of the photodiode PD to the chargeholding portion MEM2. When turned on, the transfer transistor MTX2transfers charges of the charge holding portion MEM2 to the floatingdiffusion FD.

The floating diffusion FD has a voltage in accordance with the amount ofcharges transferred from the charge holding portion MEM1 or MEM2 due tocharge-to-voltage conversion caused by the capacitance component of thefloating diffusion FD. The amplifier transistor MSF is configured suchthat the voltage VDD is supplied to the drain and a bias current issupplied to the source from a current source (not illustrated) via theselect transistor MSEL and forms an amplifier unit (source followercircuit) whose input node is the gate. Thereby, the amplifier transistorMSF outputs the signal Vvl based on the voltage of the floatingdiffusion FD to the vertical signal line 16 via the select transistorMSEL. When turned on, the reset transistor MRS resets the floatingdiffusion FD to a voltage in accordance with the voltage VDD.

In the case of the pixel 12 having a circuit configuration illustratedin FIG. 12, the control line 14 on each row includes a signal lineconnected to the gate of the transfer transistor MGS1 and a signal lineconnected to the gate of the transfer transistor MGS2. The control line14 on each row further includes a signal line connected to the gate ofthe transfer transistor MTX1 and a signal line connected to the gate ofthe transfer transistor MTX2. The control line 14 on each row furtherincludes a signal connected to the gate of the charge drain transistorMOFD, a signal line connected to the gate of the reset transistor MRS,and a signal line connected to the gate of the select transistor MSEL.

The transfer transistor MGS1 is supplied with a control signal ϕGS1 fromthe vertical scanning circuit 20 via the control line 14. The transfertransistor MGS2 is supplied with a control signal ϕGS2 from the verticalscanning circuit 20 via the control line 14. The transfer transistorMTX1 is supplied with a control signal ϕTX1 from the vertical scanningcircuit 20 via the control line 14. The transfer transistor MTX2 issupplied with a control signal ϕTX2 from the vertical scanning circuit20 via the control line 14. The charge drain transistor MOFD is suppliedwith a control signal ϕOFD from the vertical scanning circuit 20 via thecontrol line 14. The reset transistor MRS is supplied with a controlsignal ϕRES from the vertical scanning circuit 20 via the control line14. The select transistor MSEL is supplied with a control signal ϕSELfrom the vertical scanning circuit 20 via the control line 14. Theplurality of pixels 12 within the pixel unit 10 are controlled on a rowbasis by control signals ϕGS1, ϕGS2, ϕTX1, ϕTX2, ϕRES, and ϕSEL suppliedfrom the vertical scanning circuit 20. When each transistor of the pixel12 is formed of an n-channel transistor, the corresponding transistor isin an on-state when the above control signal is at a High level (Hlevel), and the corresponding transistor is in an off-state when theabove control signal is at a Low level (L level).

Next, the operation timing of the imaging element 100 will be describedby using FIG. 13. FIG. 13 is a timing diagram illustrating a readoutoperation of the imaging element in the imaging device according to thepresent embodiment.

A readout operation of a signal in a certain frame is started at timeT40, for example. Time T40 is the time when a readout operation of along-time signal of the pixel 12 on the first row in the frame isstarted.

At time T40, the control signal ϕOFD is controlled to the H level toreset the photodiode PD, and the control signal ϕOFD is then controlledto the L level to start accumulation of charges in the photodiode PD.

Next, at time T41, the control signal ϕGS1 is controlled to the H levelto transfer charges accumulated in the photodiode PD during the periodfrom time T40 to time T41 to the charge holding portion MEM1. Here, theperiod from time T40 to time T41 is the accumulation time T1corresponding to a long-time signal. The control signal ϕGS1 iscontrolled to the L level to end the transfer of charges to the chargeholding portion MEM1, and the control signal ϕOFD is then controlled tothe H level to reset the photodiode PD. The control signal ϕOFD is thencontrolled to the L level to re-start accumulation of charges in thephotodiode PD.

Next, at time T42, the control signal ϕGS2 is controlled to the H levelto transfer charges accumulated in the photodiode PD during the periodfrom time T41 to time T42 to the charge holding portion MEM2. Here, theperiod from time T41 to time T42 is the accumulation time T2corresponding to a short-time signal. The control signal ϕGS2 iscontrolled to the L level to end the transfer of charges to the chargeholding portion MEM2, and the control signal ϕOFD is then controlled tothe H level to reset the photodiode PD. The control signal ϕOFD is thencontrolled to the L level to re-start accumulation of charges in thephotodiode PD.

Next, in the period from time T42 to time T45, a readout operation of asignal based on charges held in the charge holding portion MEM1(long-time signal) and a readout operation of a signal based on chargesheld in the charge holding portion MEM2 (short-time signal) areperformed on each of the pixels 12 sequentially on a row basis. Forexample, the period from time T42 to time T43 is a period in which asignal based on charges accumulated in the charge holding portion MEM1of the pixel 12 on the first row (long-time signal) is read out. Theperiod from time T43 to time T44 is a period in which a signal based oncharges accumulated in the charge holding portion MEM2 of the pixel 12on the first row (short-time signal) is read out. Readout operations ofa long-time signal and a short-time signal from the pixel 12 on thesecond and subsequent rows are performed sequentially on a row basis onand after time T44.

In a readout operation of a long-time signal, after the floatingdiffusion FD is reset, charges accumulated in the charge holding portionMEM1 are transferred to the floating diffusion FD, and a signal inaccordance with the voltage of the floating diffusion FD is output tothe vertical signal line 16. In a readout operation of a short-timesignal, after the floating diffusion FD is reset, charges accumulated inthe charge holding portion MEM2 are transferred to the floatingdiffusion FD, and a signal in accordance with the voltage of thefloating diffusion FD is output to the vertical signal line 16. Such aseries of readout operations are performed for respective rows under thecontrol of the vertical scanning circuit 20, and the time of the end ofa readout operation of the pixel 12 on the m-th row, which is the lastrow, is time T45.

Next, a readout operation for one row in the imaging element 100 will bedescribed more specifically by using FIG. 14. FIG. 14 is a timingdiagram illustrating a readout operation for one row of the imagingelement in the imaging device according to the present embodiment.

As described previously, in a pixel on the first row, the period fromtime T42 to time T43 is a readout period for a long-time signalcorresponding to the accumulation time T1, and the period from time T43to time T44 is a readout period for a short-time signal corresponding tothe accumulation time T2. Note that, while the example in which along-time signal is read out and then a short-time signal is read out isdescribed here, a short-time signal may be read out and then a long-timesignal may be read out.

First, an operation from time T42 to time T43, which is a readout periodfor a long-time signal of the pixel 12 on the first row, will bedescribed.

In the initial state where the pixel 12 on the first row is selected,the control signal ϕRES is temporarily controlled to the H level by thevertical scanning circuit 20, and the voltage of the floating diffusionFD of the pixel 12 is reset. Thereby, a signal (N-signal) in accordancewith the reset voltage of the floating diffusion FD is output to thevertical signal line 16 as the signal Vvl. Note that the N-signalcorresponding to a long-time signal is here referred to as “N1-signal”.

Further, in parallel to a reset operation of the floating diffusion FD,the reset signal of the amplifier 33 is temporarily controlled to the Hlevel, and the output of the amplifier 33 (signal Vamp) is reset. Afterthe control signal ϕRES is controlled to the L level by the verticalscanning circuit 20 and the reset operation of the floating diffusion FDof the pixel 12 ends, the reset signal of the amplifier 33 is controlledto the L level and the reset operation of the amplifier 33 ends.Thereby, the N1-signal amplified at a predetermined gain (one-fold inthis example) set in the amplifier 33 is output from the amplifier 33 asthe signal Vamp.

Next, an AD conversion process is performed on the signal Vamp outputfrom the amplifier 33. Hereafter, AD conversion performed on theN1-signal is referred to as “N1 conversion”. The reference signalgeneration circuit 36 starts changing the reference signal Vr at aconstant rate. Further, the counter 46 starts counting insynchronization with the start of a change in the reference signal Vrand outputs a count signal indicating a count value to the memory 42 oneach column.

The comparator 35 compares the signal level of the signal Vamp with thesignal level of the reference signal Vr and outputs a latch signal Vcmpto the memory 42 at a timing when the compared signal levels become thesame. The memory 42 latches the count value received from the counter 46at a timing of receiving the latch signal Vcmp from the comparator 35and holds the value in the N-memory 44-2. The count value held in theN-memory 44-2 corresponds to the digital N1-signal obtained byperforming AD conversion on the N1-signal.

Next, the control signal ϕTX1 is temporarily controlled to the H levelby the vertical scanning circuit 20, and charges held in the chargeholding portion MEM1 are transferred to the floating diffusion FD.Thereby, a signal (S-signal) in accordance with the amount of chargestransferred to the floating diffusion FD is output to the verticalsignal line 16 as the signal Vvl. Note that the S-signal correspondingto the long-time signal is here referred to as “S1-signal”. TheS1-signal amplified at a predetermined gain (one-fold in this example)set in the amplifier 33 is then output from the amplifier 33 as thesignal Vamp. Thereby, the output voltage of the signal Vamp increases bya voltage ΔV1 corresponding to the S1-signal.

Next, an AD conversion process is performed on the signal Vamp outputfrom the amplifier 33. Hereafter, AD conversion performed on theS1-signal is referred to as “S1 conversion”. The reference signalgeneration circuit 36 starts changing the reference signal Vr at aconstant rate. Further, the counter 46 starts counting insynchronization with the start of a change in the reference signal Vrand outputs a count value to the memory 42 on each column.

The comparator 35 compares the signal level of the signal Vamp with thesignal level of the reference signal Vr and outputs a latch signal Vcmpto the memory 42 at a timing when the compared signal levels become thesame. The memory 42 latches the count value received from the counter 46at a timing of receiving the latch signal Vcmp from the comparator 35and holds the value in the S-memory 44-3. The count value held in theS-memory 44-3 corresponds to the digital S1-signal obtained byperforming AD conversion on the S1-signal.

On the other hand, when the signal level of the signal Vamp is greaterthan or equal to a preset saturation level and Vamp is always greaterthan Vr during the S1 conversion period, the long-time signal saturationdecision unit 38 outputs, to the memory 42, the saturation decisionsignal JL that has the H level at the end of the S1 conversion period.The saturation decision signal JL is held in the JL memory 44-1 of thememory 42.

Next, an operation from time T43 to time T44, which is a readout periodfor a short-time signal of the pixel 12 on the first row, will bedescribed.

The state where the pixel 12 on the first row is selected continues fromthe readout period of a long-time signal. In this state, the controlsignal ϕRES is temporarily controlled to the H level by the verticalscanning circuit 20, and the voltage of the floating diffusion FD of thepixel 12 is reset. Thereby, a signal (N-signal) in accordance with thereset voltage of the floating diffusion FD is output to the verticalsignal line 16 as the signal Vvl. Note that the N-signal correspondingto a short-time signal is here referred to as “N2-signal”.

Further, in parallel to a reset operation of the floating diffusion FD,the reset signal of the amplifier 33 is temporarily controlled to the Hlevel, and the output of the amplifier 33 (signal Vamp) is reset. Afterthe control signal ϕRES is controlled to the L level by the verticalscanning circuit 20 and the reset operation of the floating diffusion FDof the pixel 12 ends, the reset signal of the amplifier 33 is controlledto the L level and the reset operation of the amplifier 33 ends.Thereby, the N2-signal amplified at a predetermined gain (one-fold inthis example) set in the amplifier 33 is output from the amplifier 33 asthe signal Vamp.

Next, an AD conversion process is performed on the signal Vamp outputfrom the amplifier 33. Hereafter, AD conversion performed on theN2-signal is referred to as “N2 conversion”. The reference signalgeneration circuit 36 starts changing the reference signal Vr at aconstant rate. Further, the counter 46 starts counting insynchronization with the start of a change in the reference signal Vrand outputs a count signal indicating a count value to the memory 42 oneach column.

The comparator 35 compares the signal level of the signal Vamp with thesignal level of the reference signal Vr and outputs a latch signal Vcmpto the memory 42 at a timing when the compared signal levels become thesame. The memory 42 latches the count value received from the counter 46at a timing of receiving the latch signal Vcmp from the comparator 35and holds the value in the N-memory 44-4. The count value held in theN-memory 44-4 corresponds to the digital N2-signal obtained byperforming AD conversion on the N2-signal.

Next, the control signal ϕTX2 is temporarily controlled to the H levelby the vertical scanning circuit 20, and charges held in the chargeholding portion MEM2 are transferred to the floating diffusion FD.Thereby, a signal (S-signal) in accordance with the amount of chargestransferred to the floating diffusion FD is output to the verticalsignal line 16 as the signal Vvl. Note that the S-signal correspondingto the short-time signal is here referred to as “S2-signal”. TheS2-signal amplified at a predetermined gain (one-fold in this example)set in the amplifier 33 is output from the amplifier 33 as the signalVamp. Thereby, the output voltage of the signal Vamp increases by avoltage ΔV2 corresponding to the S2-signal.

Next, an AD conversion process is performed on the signal Vamp outputfrom the amplifier 33. Hereafter, AD conversion performed on theS2-signal is referred to as “S2 conversion”. The reference signalgeneration circuit 36 starts changing the reference signal Vr at aconstant rate. Further, the counter 46 starts counting insynchronization with the start of a change in the reference signal Vrand outputs a count signal indicating a count value to the memory 42 oneach column.

The comparator 35 compares the signal level of the signal Vamp with thesignal level of the reference signal Vr and outputs a latch signal Vcmpto the memory 42 at a timing when the compared signal levels become thesame. The memory 42 latches the count value received from the counter 46at a timing of receiving the latch signal Vcmp from the comparator 35and holds the value in the S-memory 44-5. The count value held in theS-memory 44-5 corresponds to the digital S2-signal obtained byperforming AD conversion on the S2-signal.

In such a way, the saturation decision signal JL, the digital N1-signal,the digital S1-signal, the digital N2-signal, and the digital S2-signalare held in the memory 42 on each column. These digital signals held inthe memory 42 on each column are transferred to the signal processingunit 60 sequentially on a column basis in accordance with a controlsignal supplied from the horizontal scanning circuit 50.

Next, the configuration and the operation of the signal processing unit200 will be described by using FIG. 15. FIG. 15 is a diagramillustrating the configuration and the operation of the signalprocessing unit 200 in the imaging device according to the presentembodiment.

As illustrated in FIG. 15, the signal processing unit 200 of the imagingdevice according to the present embodiment includes the signal selectunit 250 and the HDR composition unit 260. The signal select unit 250includes the long/short select signal generation unit 252 and theselector 254.

The long/short select signal generation unit 252 holds the saturationdecision signals JL received from the imaging element 100 for aplurality of pixels. In a case of a monochrome sensor, the long/shortselect signal generation unit 252 holds a saturation decision signalJL(N−1) corresponding to the pixel 12 on the (N−1)-th column, asaturation decision signal JL(N) corresponding to the pixel 12 on theN-th column, and a saturation decision signal JL(N+1) corresponding tothe pixel 12 on the (N+1)-th column. Further, the long/short selectsignal generation unit 252 generates a long/short select signal JJ(N)corresponding to the pixel 12 on the N-th column based on the saturationdecision signals JL(N−1), JL(N), and JL(N+1) and transmits the generatedlong/short select signal JJ(N) to the selector 254.

If the saturation decision signal JL(N) is 1, a long/short select signalJJ(N) of 1 is output unconditionally. Even if the saturation decisionsignal JL(N) is 0, if one of the saturation decision signal JL(N−1) andthe saturation decision signal JL(N+1) is 1, a long/short select signalJJ(N) of 1 is output. Only if all the saturation decision signalsJL(N−1), JL(N), and JL(N+1) are 0, a long/short select signal JJ(N) of 0is output.

The selector 254 selects and outputs one of the long-time signal (N) andthe short-time signal (N) received from the imaging element 100 inaccordance with the long/short select signal JJ(N) received from thelong/short select signal generation unit 252. In this example, thelong-time signal (N) corresponds to the digital N1-signal transferredfrom the N-memory 44-2 and the digital S1-signal transferred from theS-memory 44-3. Further, the short-time signal (N) corresponds to thedigital N2-signal transferred from the N-memory 44-4 and the digitalS2-signal transferred from the S-memory 44-5. For example, if thelong/short select signal JJ(N) is 0, the long-time signal (N) is output,and if the long/short select signal JJ(N) is 1, the short-time signal isoutput.

In such a way, the signal select unit 250 outputs, to the HDRcomposition unit 260, the long/short select signal JJ(N) generated bythe long/short select signal generation unit 252 and a signal selectedby the selector 254 out of a long-time signal (N) and a short-timesignal (N).

After removing a noise component by performing a calculation process tosubtract the N-signal from the S-signal, the HDR composition unit 260performs an HDR composition process and an HDR compression process andoutputs HDR compression data by using the scheme described in the firstembodiment, for example.

Also in the imaging device according to the present embodiment, as withthe first embodiment, even with a signal of the pixel in which thelong-time signal is not saturated, when the long-time signal of theadjacent pixels is saturated, a pixel value is calculated by using thevalue of the short-time signal. Therefore, an HDR composition signalhaving small spatial variation can be obtained also by the imagingdevice of the present embodiment. Furthermore, since saturation decisionprocesses are performed in parallel with respect to columns inside theimaging device in the present embodiment, a faster decision process ispossible. Further, since a signal select process is performed inside theimaging device, a band of data transmitted to the outside of the imagingdevice can also be reduced.

As described above, according to the present embodiment, when composinga high dynamic range image by using a long-time image and a short-timeimage, image quality at a joining part between the long-time image andthe short-time image can be improved.

Fourth Embodiment

An imaging device according to a fourth embodiment of the presentinvention will be described with reference to FIG. 16 to FIG. 18B. Thesame component as that of the imaging device according to the first tothird embodiments is labeled with the same reference, and thedescription thereof will be omitted or simplified.

First, the structure of the imaging device according to the presentembodiment will be described by using FIG. 16. FIG. 16 is a blockdiagram illustrating a general configuration of an imaging element inthe imaging device in accordance with the present embodiment. Note thatthe entire configuration of the imaging device according to the presentembodiment is basically the same as the imaging device according to thefirst embodiment illustrated in FIG. 1.

As illustrated in FIG. 16, the imaging element 100 of the imaging deviceaccording to the present embodiment includes a signal select unit 48corresponding to the signal select unit 250 in the first to thirdembodiments on each column. Further, the JL-memory 44-1 is providedseparately from the memory 42 and connected to the memory 42 via thesignal select unit 48. Further, the memories 42 share the N-memory 44-2as a memory used for holding the N1-signal of a long-time signal and theN2-signal of a short-time signal and share the S-memory 44-3 as a memoryused for holding the S1-signal of a long-time signal and the S2-signalof a short-time signal. The memory 42 further includes a JJ-memory 44-6.Other features are basically the same as the imaging element of theimaging device according to the third embodiment illustrated in FIG. 11.

The signal Vvl output from each of the pixels 12 is read out to thevertical signal line 16 on the corresponding column on a row basis bythe vertical scanning circuit 20. The amplifier 33 on each columnamplifies the signal Vvl read out from the pixel 12 via the verticalsignal line 16 on the corresponding column at a predetermined gain andoutputs the amplified signal Vamp to the comparator 35 and the long-timesignal saturation decision unit 38. The long-time signal saturationdecision unit 38 on each column decides whether or not the level of asignal amplified by the amplifier 33 on the corresponding column exceedsthe level of a preset predetermined saturation threshold value andoutputs the saturation decision signal JL indicating a decision resultto the JL-memory 44-1 on the corresponding column. The JL-memory 44-1holds the saturation decision signal JL received from the long-timesignal saturation decision unit 38.

The reference signal generation circuit 36 outputs, to the comparator35, a reference signal Vr whose signal level changes at a constant ratewith time. The comparator 35 on each column compares the signal level ofsignal Vamp from the amplifier 33 on the corresponding column with thesignal level of the reference signal Vr and outputs the signal Vcmpindicating a comparison result to the memory 42 and the signal selectunit 48. The counter 46 outputs a count signal indicating a count valuethat is a counting value of the clock signal to the memory 42 on eachcolumn.

The signal select unit 48 generates the long/short select signal JJbased on the latch signal Vcmp received from the comparator 35 and thesaturation decision signal JL held in the JL-memory 44-1 and outputs thegenerated long/short select signal JJ to the memory 42. The memory 42holds the long/short select signal JJ generated by the signal selectunit 48 in the JJ-memory 44-6.

The memory 42 on each column holds, as a digital signal obtained byperforming AD conversion on the signal Vcmp, a count signal receivedfrom the counter 46 when the signal level of the signal Vcmp changes.The memory 42 holds a digital signal obtained after AD conversion in theN-memory 44-2 or the S-memory 44-3 in accordance with the type of thesignal.

The horizontal scanning circuit 50 reads out the long/short selectsignal JJ and the digital signals N and S held in the memory 42 on eachcolumn sequentially on a column basis and transfers the read out signalto the signal processing unit 60. The signal processing unit 60 performsa calculation process on the long/short select signal JJ and the digitalsignals N and S sequentially read out from the memory 42 and outputs thedigital signal D.

Next, a readout operation for one row in the imaging element 100 will bedescribed by using FIG. 17. FIG. 17 is a timing diagram illustrating areadout operation for one row of the imaging element in the imagingdevice according to the present embodiment.

First, an operation from time T52 to time T53, which is a readout periodfor a long-time signal of the pixel 12 on the first row, will bedescribed.

In the initial state where the pixel 12 on the first row is selected,the control signal ϕRES is temporarily controlled to the H level by thevertical scanning circuit 20, and the voltage of the floating diffusionFD of the pixel 12 is reset. Thereby, a signal (N1-signal) in accordancewith the reset voltage of the floating diffusion FD is output to thevertical signal line 16 as the signal Vvl.

Further, in parallel to a reset operation of the floating diffusion FD,the reset signal of the amplifier 33 is temporarily controlled to the Hlevel, and the output of the amplifier 33 (signal Vamp) is reset. Afterthe control signal ϕRES is controlled to the L level by the verticalscanning circuit 20 and the reset operation of the floating diffusion FDof the pixel 12 ends, the reset signal of the amplifier 33 is controlledto the L level and the reset operation of the amplifier 33 ends.Thereby, the N1-signal amplified at a predetermined gain (one-fold inthis example) set in the amplifier 33 is output from the amplifier 33 asthe signal Vamp.

Next, the AD conversion process (N1 conversion) is performed on thesignal Vamp output from the amplifier 33. The reference signalgeneration circuit 36 starts changing the reference signal Vr at aconstant rate. Further, the counter 46 starts counting insynchronization with the start of a change in the reference signal Vrand outputs a count signal indicating a count value to the memory 42 oneach column.

The comparator 35 compares the signal level of the signal Vamp with thesignal level of the reference signal Vr and outputs a latch signal Vcmpto the memory 42 at a timing when the compared signal levels become thesame. The memory 42 latches the count value received from the counter 46at a timing of receiving the latch signal Vcmp from the comparator 35and holds the value in the N-memory 44-2. The count value held in theN-memory 44-2 corresponds to the digital N1-signal obtained byperforming AD conversion on the N1-signal.

Next, the control signal ϕTX1 is temporarily controlled to the H levelby the vertical scanning circuit 20, and charges held in the chargeholding portion MEM1 are transferred to the floating diffusion FD.Thereby, a signal (S1-signal) in accordance with the amount of chargestransferred to the floating diffusion FD is output to the verticalsignal line 16 as the signal Vvl. The S1-signal amplified at apredetermined gain (one-fold in this example) set in the amplifier 33 isthen output from the amplifier 33 as the signal Vamp. Thereby, theoutput voltage of the signal Vamp increases by the voltage ΔV1corresponding to the S1-signal.

Next, the AD conversion process (S1 conversion) is performed on thesignal Vamp output from the amplifier 33. The reference signalgeneration circuit 36 starts changing the reference signal Vr at aconstant rate. Further, the counter 46 starts counting insynchronization with the start of a change in the reference signal Vrand outputs a count value to the memory 42 on each column.

The comparator 35 compares the signal level of the signal Vamp with thesignal level of the reference signal Vr and outputs a latch signal Vcmpto the memory 42 at a timing when the compared signal levels become thesame. The memory 42 latches the count value received from the counter 46at a timing of receiving the latch signal Vcmp from the comparator 35and holds the value in the S-memory 44-3. The count value held in theS-memory 44-3 corresponds to the digital S1-signal obtained byperforming AD conversion on the S1-signal.

On the other hand, when the signal level of the signal Vamp is greaterthan or equal to a preset saturation level and Vamp is always greaterthan Vr during the S1 conversion period, the long-time signal saturationdecision unit 38 outputs the saturation decision signal JL having the Hlevel at the end of the S1 conversion period to the JL-memory 44-1.

Next, in the period from time T53 to time T54, the long/short selectsignal JJ used as a criterion for whether a long-time signal is held ora short-time signal is held in the N-memory 44-2 and the S-memory 44-3is generated. The signal select unit 48 generates the long/short selectsignal JJ based on the saturation decision signal JL and holds thegenerated long/short select signal JJ in the JJ-memory 44-6 of thememory 42. For example, in a case of a monochrome sensor, the signalselect unit 48 generates a long/short select signal JJ(N) correspondingto the pixel 12 on the N-th column based on the saturation decisionsignals JL(N−1), JL(N), and JL(N+1) and holds the generated long/shortselect signal JJ(N) in the JJ-memory 44-6 on the N-th column. If thesaturation decision signal JL(N) is 1, a long/short select signal JJ(N)of 1 is output unconditionally. Even if the saturation decision signalJL(N) is 0, if one of the saturation decision signal JL(N−1) and thesaturation decision signal JL(N+1) is 1, a long/short select signalJJ(N) of 1 is output. Only if all the saturation decision signalsJL(N−1), JL(N), and JL(N+1) are 0, the long/short select signal JJ(N) of0 is output.

For example, when holding a short-time signal in the N-memory 44-2 andthe S-memory 44-3, the signal select unit 48 outputs the long/shortselect signal JJ(N) of the H level. When holding a long-time signal inthe N-memory 44-2 and the S-memory 44-3, the signal select unit 48outputs the long/short select signal JJ(N) of the L level.

Next, an operation from time T54 to time T55, which is a readout periodfor a short-time signal of the pixel 12 on the first row, will bedescribed.

The state where the pixel 12 on the first row is selected continues fromthe readout period of a long-time signal. In this state, the controlsignal ϕRES is temporarily controlled to the H level by the verticalscanning circuit 20, and the voltage of the floating diffusion FD of thepixel 12 is reset. Thereby, a signal (N2-signal) in accordance with thereset voltage of the floating diffusion FD is output to the verticalsignal line 16 as the signal Vvl.

Further, in parallel to a reset operation of the floating diffusion FD,the reset signal of the amplifier 33 is temporarily controlled to the Hlevel, and the output of the amplifier 33 (signal Vamp) is reset. Afterthe control signal ϕRES is controlled to the L level by the verticalscanning circuit 20 and the reset operation of the floating diffusion FDof the pixel 12 ends, the reset signal of the amplifier 33 is controlledto the L level and the reset operation of the amplifier 33 ends.Thereby, the N2-signal amplified at a predetermined gain (one-fold inthis example) set in the amplifier 33 is output from the amplifier 33 asthe signal Vamp.

Next, the AD conversion process (N2 conversion) is performed on thesignal Vamp output from the amplifier 33. The reference signalgeneration circuit 36 starts changing the reference signal Vr at aconstant rate. Further, the counter 46 starts counting insynchronization with the start of a change in the reference signal Vrand outputs a count signal indicating a count value to the memory 42 oneach column.

The comparator 35 compares the signal level of the signal Vamp with thesignal level of the reference signal Vr and outputs a latch signal Vcmpto the signal select unit 48 at a timing when the compared signal levelsbecome the same. The signal select unit 48 generates a write signalVcmp2 in accordance with the latch signal Vcmp and the long/short selectsignal JJ held in the JJ-memory 44-6 and outputs the generated writesignal Vcmp2 to the memory 42. The memory 42 latches the count valuereceived from the counter 46 at a timing of receiving the write signalVcmp2 of the H level from the signal select unit 48 and holds the valuein the N-memory 44-2.

Specifically, the signal select unit 48 outputs the write signal Vcmp2of the H level if both the latch signal Vcmp and the long/short selectsignal JJ are at the H level. That is, if the long/short select signalJJ is at the H level, the memory 42 holds a count value (digitalN2-signal) in accordance with the output timing of the latch signal Vcmpin the N-memory 44-2. On the other hand, if the long/short select signalJJ is at the L level, the digital N2-signal is not held in the N-memory44-2.

Next, the control signal ϕTX2 is temporarily controlled to the H levelby the vertical scanning circuit 20, and charges held in the chargeholding portion MEM2 are transferred to the floating diffusion FD.Thereby, a signal (S2-signal) in accordance with the amount of chargestransferred to the floating diffusion FD is output to the verticalsignal line 16 as the signal Vvl. The S2-signal amplified at apredetermined gain (one-fold in this example) set in the amplifier 33 isthen output from the amplifier 33 as the signal Vamp. Thereby, theoutput voltage of the signal Vamp increases by the voltage AV2corresponding to the S2-signal.

Next, the AD conversion process (S2 conversion) is performed on thesignal Vamp output from the amplifier 33. The reference signalgeneration circuit 36 starts changing the reference signal Vr at aconstant rate. Further, the counter 46 starts counting insynchronization with the start of a change in the reference signal Vrand outputs a count signal indicating a count value to the memory 42 oneach column.

The comparator 35 compares the signal level of the signal Vamp with thesignal level of the reference signal Vr and outputs a latch signal Vcmpto the signal select unit 48 at a timing when the compared signal levelsbecome the same. The signal select unit 48 generates a write signalVcmp2 in accordance with the latch signal Vcmp and the long/short selectsignal JJ held in the JJ-memory 44-6 and outputs the generated writesignal Vcmp2 to the memory 42. The memory 42 latches the count valuereceived from the counter 46 at a timing of receiving the write signalVcmp2 of the H level from the signal select unit 48 and holds the valuein the S-memory 44-3.

Specifically, the signal select unit 48 outputs the write signal Vcmp2of the H level if both the latch signal Vcmp and the long/short selectsignal JJ are at the H level. That is, if the long/short select signalJJ is at the H level, the memory 42 holds a count value (digitalS2-signal) in accordance with the output timing of the latch signal Vcmpin the S-memory 44-3. On the other hand, if the long/short select signalJJ is at the L level, the digital S2-signal is not held in the S-memory44-3.

As a result, if the long/short select signal JJ is at the H level, thedigital N2-signal is held in the N-memory 44-2 as the digital N-signal,and the digital S2-signal is held in the S-memory 44-3 as the digitalS-signal. Further, if the long/short select signal JJ is at the L level,the digital N1-signal is held in the N-memory 44-2 as the digitalN-signal, and the digital S1-signal is held in the S-memory 44-3 as thedigital S-signal.

The long/short select signal JJ, the digital N-signal, and the digitalS-signal held in the memory 42 on each column are transferred to thesignal processing unit 60 sequentially on a column basis in accordancewith a control signal supplied from the horizontal scanning circuit 50.The signal processing unit 60 performs the same process as the HDRcomposition unit 260 in the third embodiment and outputs HDR compressiondata D.

Next, a configuration example of the signal select unit 48 will bedescribed by using FIG. 18A and FIG. 18B. FIG. 18A and FIG. 18B arecircuit diagrams illustrating a configuration example of the signalselect unit 48. FIG. 18A is a generation circuit of the long/shortselect signal JJ, and FIG. 18B is a generation circuit of the writesignal Vcmp2.

The signal select unit 48 includes a generation circuit of thelong/short select signal JJ and a generation circuit of the write signalVcmp2. For example, as illustrated in FIG. 18A, the generation circuitof the long/short select signal JJ is formed of a plurality of ORcircuits, each of which calculates a logical product of the saturationdecision signals JL for every adjacent three columns. For example, thelogical product of the saturation decision signals JL(N−1), JL(N), andJL(N+1) corresponds to the long/short select signal JJ(N) on the N-thcolumn. For example, as illustrated in FIG. 18B, the generation circuitof the write signal Vcmp2 is formed of an AND circuit that calculates alogical sum of the long/short select signal JJ and the latch signalVcmp. For example, the logical product of the long/short select signalJJ(N) and the latch signal Vcmp(N) corresponds to the write signalVcmp2(N) on the N-th column.

Also in the imaging device according to the present embodiment, as withthe first embodiment, even with a signal of the pixel in which thelong-time signal is not saturated, when the long-time signal of theadjacent pixels is saturated, a pixel value is calculated by using thevalue of the short-time signal. Therefore, an HDR composition signalhaving small spatial variation can be obtained also by the imagingdevice of the present embodiment. Furthermore, since saturation decisionoperations and signal select operations are performed in parallel withrespect to columns inside the imaging element in the present embodiment,a faster decision process is possible. Further, since it is notnecessary to provide column memories in association with each of thelong-time signal and a short-time signal, the area of a peripheralcircuit of the imaging element can be reduced. Further, since a signalselect operation is performed inside the imaging device, a band of datatransmitted to the outside of the imaging device can also be reduced.

As described above, according to the present embodiment, when composinga high dynamic range image by using a long-time image and a short-timeimage, image quality at a joining part between the long-time image andthe short-time image can be improved.

Fifth Embodiment

An imaging system according to a fifth embodiment of the presentinvention will be described with reference to FIG. 19. FIG. 19 is ablock diagram illustrating a general configuration of the imaging systemaccording to the present embodiment.

The imaging device 300 described in the first to fourth embodimentsdescribed above can be applied to various imaging systems. Examples ofapplicable imaging systems may include a digital still camera, a digitalcamcorder, a surveillance camera, a copying machine, a fax machine, amobile phone, an on-vehicle camera, an observation satellite, and thelike. In addition, a camera module including an optical system such as alens and an imaging device is also included in the imaging system. FIG.19 illustrates a block diagram of a digital still camera as an exampleout of these examples.

An imaging system 400 illustrated as an example in FIG. 19 includes animaging device 401, a lens 402 that captures an optical image of anobject onto the imaging device 401, an aperture 404 for changing a lightamount passing through the lens 402, and a barrier 406 for protectingthe lens 402. The lens 402 and the aperture 404 form an optical systemthat converges a light onto the imaging device 401. The imaging device401 is the imaging device 300 described in any of the first to fourthembodiments and converts an optical image captured by the lens 402 intoimage data.

Further, the imaging system 400 includes a signal processing unit 408that processes an output signal output from the imaging device 401. Thesignal processing unit 408 preforms an operation of signal processing toperform various correction or compression on an input signal, ifnecessary, and output the processed signal. For example, the signalprocessing unit 408 performs predetermined image processing such as aconversion process to convert RGB pixel output signals into aY, Cb, Crcolor space, gamma correction, or the like on the input signal. Further,the signal processing unit 408 may have a part or all of the function ofthe signal processing unit 200 in the imaging device 300 described inthe first to fourth embodiments.

Furthermore, the imaging system 400 includes a memory unit 410 fortemporarily storing image data therein and an external interface unit(external I/F unit) 412 for communicating with an external computer orthe like. The imaging system 400 further includes a storage medium 414such as a semiconductor memory for performing storage or readout ofimaging data and a storage medium control interface unit (storage mediumcontrol I/F unit) 416 for performing storage or readout on the storagemedium 414. Note that the storage medium 414 may be embedded in theimaging system 400 or may be removable.

Furthermore, the imaging system 400 includes a general control/operationunit 418 that performs various calculation and controls the entiredigital still camera and a timing generation unit 420 that outputsvarious timing signals to the imaging device 401 and the signalprocessing unit 408. Here, the timing signal or the like may be inputfrom the outside, and the imaging system 400 may include at least theimaging device 401 and the signal processing unit 408 that processes anoutput signal output from the imaging device 401. The generalcontrol/operation unit 418 and the timing generation unit 420 may beconfigured to implement a part or all of the control function of theimaging device 401.

The imaging device 401 outputs an imaging signal to the signalprocessing unit 408. The signal processing unit 408 performspredetermined signal processing on an imaging signal output from theimaging device 401 and outputs image data. The signal processing unit408 uses an imaging signal to generate an image. The image generated bythe signal processing unit 408 is stored in the storage medium 414, forexample. Further, an image generated by the signal processing unit 408is projected on a monitor formed of a liquid crystal display or the likeas a moving image or a static image. An image stored in the storagemedium 414 can be hard-copied by a printer or the like.

Note that the function provided by the signal processing unit 408 may beimplemented by a signal processing device other than the imaging device300. Further, the imaging device 300 may have the function provided bythe signal processing unit 408.

As discussed above, according to the present embodiment, the imagingsystem to which the imaging device 300 according to the first to fourthembodiment is applied can be realized.

Sixth Embodiment

An imaging system and a movable object according to a sixth embodimentof the present invention will be described with reference to FIG. 20Aand FIG. 20B. FIG. 20A is a schematic diagram illustrating aconfiguration example of the imaging system according to the presentembodiment. FIG. 20B is a schematic diagram illustrating a configurationexample of the movable object according to the present embodiment.

FIG. 20A illustrates an example of an imaging system related to anon-vehicle camera. The imaging system 500 includes an imaging device510. The imaging device 510 is the imaging device 300 described in anyof the above first to fourth embodiments. The imaging system 500includes an image processing unit 512 that performs image processing ona plurality of image data acquired by the imaging device 510 and aparallax acquisition unit 514 that calculates a parallax (a phasedifference of parallax images) from the plurality of image data acquiredby the imaging device 510. Further, the imaging system 500 includes adistance acquisition unit 516 that calculates a distance to the objectbased on the calculated parallax and a collision determination unit 518that determines whether or not there is a collision possibility based onthe calculated distance. Here, the parallax acquisition unit 514 and thedistance acquisition unit 516 are an example of a distance informationacquisition unit that acquires distance information on the distance tothe object. That is, the distance information is information on aparallax, a defocus amount, a distance to an object, or the like. Thecollision determination unit 518 may use any of the distance informationto determine the collision possibility. The distance informationacquisition unit may be implemented by dedicatedly designed hardware ormay be implemented by a software module. Further, the distanceinformation acquisition unit may be implemented by a Field ProgrammableGate Array (FPGA), an Application Specific Integrated Circuit (ASIC), orthe like or may be implemented by a combination thereof.

The imaging system 500 is connected to the vehicle informationacquisition device 520 and can acquire vehicle information such as avehicle speed, a yaw rate, a steering angle, or the like. Further, theimaging system 500 is connected to a control ECU 530, which is a controldevice that outputs a control signal for causing a vehicle to generatebraking force based on a determination result by the collisiondetermination unit 518. That is, the control ECU 530 is an example of amovable object control unit that controls a movable object based ondistance information. Further, the imaging system 500 is also connectedto an alert device 540 that issues an alert to the driver based on adetermination result by the collision determination unit 518. Forexample, when the collision probability is high as the determinationresult of the collision determination unit 518, the control ECU 530performs vehicle control to avoid a collision or reduce damage byapplying a brake, pushing back an accelerator, suppressing engine power,or the like. The alert device 540 alerts a user by sounding an alertsuch as a sound, displaying alert information on a display of a carnavigation system or the like, providing vibration to a seat belt or asteering wheel, or the like.

In the present embodiment, an area around a vehicle, for example, afront area or a rear area is captured by using the imaging system 500.FIG. 20B illustrates the imaging system 500 when a front area of avehicle (a capturing area 550) is captured. The vehicle informationacquisition device 520 transmits an instruction to cause the imagingsystem 500 to operate and perform capturing. Such a configuration canfurther improve the ranging accuracy. With the use of the imaging deviceof the first to fourth embodiments as the imaging device 510, theimaging system 500 of the present embodiment can further improve rangingaccuracy.

Although the example of control for avoiding a collision to anothervehicle has been described above, the embodiment is applicable toautomatic driving control for following another vehicle, automaticdriving control for not going out of a traffic lane, or the like.Furthermore, the imaging system is not limited to a vehicle such as thesubject vehicle and can be applied to a movable object (movingapparatus) such as a ship, an airplane, or an industrial robot, forexample. In addition, the imaging system can be widely applied to adevice which utilizes object recognition, such as an intelligenttransportation system (ITS), without being limited to movable objects.

Modified Embodiments

The present invention is not limited to the embodiments described above,and various modifications are possible.

For example, an example in which a part of the configuration of any ofthe embodiments is added to another embodiment or an example in which apart of the configuration of any of the embodiments is replaced with apart of the configuration of another embodiment is also one of theembodiments of the present invention.

Further, the imaging systems illustrated in the above fifth and sixthembodiments are examples of an imaging system to which the imagingdevice of the present invention may be applied, and an imaging system towhich the imaging device of the present invention can be applied is notlimited to the configuration illustrated in FIG. 19 and FIG. 20A.

The present invention may be realized also by a process in which aprogram that implements one or more functions of the embodimentsdescribed above is supplied to a system or a device via a network or astorage medium and then one or more processors in a computer of thesystem or the device read out and execute the program. Further, thepresent invention may be realized by a circuit (for example, ASIC) thatimplements one or more functions.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2018-201644, filed Oct. 26, 2018 which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An imaging device comprising: a plurality ofpixels arranged over a plurality of rows and a plurality of columns andeach having a photoelectric converter; a control unit that controls theplurality of pixels so as to output, from each of the plurality ofpixels, a first signal based on charges generated by the photoelectricconverter during a first exposure time and a second signal based oncharges generated by the photoelectric converter during a secondexposure time that is shorter than the first exposure time; a decisionunit that performs a decision process to decide whether or not apredetermined value is exceeded for the first signal output from each ofthe plurality of pixels and outputs a decision signal indicating adecision result; and a signal select unit that selects one of the firstsignal and the second signal as an image forming signal of each of theplurality of pixels, wherein based on the decision signal of a targetpixel and the decision signal of another pixel arranged in apredetermined region including the target pixel, the signal select unitselects the image forming signal of the target pixel.
 2. The imagingdevice according to claim 1, wherein the predetermined region includes aplurality of columns adjacent to each other.
 3. The imaging deviceaccording to claim 1, wherein the predetermined region includes aplurality of rows adjacent to each other.
 4. The imaging deviceaccording to claim 1, wherein the another pixel is a pixel adjacent tothe target pixel.
 5. The imaging device according to claim 1, whereinthe another pixel is a pixel comprising a color filter of the same coloras the target pixel.
 6. The imaging device according to claim 1, whereinthe signal select unit selects the first signal as the image formingsignal of the target pixel when the decision signal of the target pixeland the decision signal of the another pixel indicate that the firstsignal does not exceed the predetermined value, and selects the secondsignal as the image forming signal of the target pixel when the decisionsignal of the another pixel indicates that the first signal exceeds thepredetermined value.
 7. The imaging device according to claim 1, whereinthe predetermined value is a saturation threshold value of the firstsignal.
 8. The imaging device according to claim 1 further comprising aplurality of memories provided in association with the plurality ofcolumns, respectively, wherein each of the plurality of memories holdsthe first signal and the second signal output from the pixels arrangedon a corresponding column.
 9. The imaging device according to claim 1further comprising a plurality of memories provided in association withthe plurality of columns, respectively, wherein each of the plurality ofmemories holds a signal selected as the image forming signal by thesignal select unit from the first signal and the second signal outputfrom the pixels arranged on a corresponding column.
 10. The imagingdevice according to claim 9, wherein the signal select unit performs, inparallel with respect to columns, a signal select process on signalsoutput from the pixels on each column.
 11. The imaging device accordingto claim 1, wherein the decision unit performs, in parallel with respectto columns, the decision process on the first signal output from thepixels on each column.
 12. The imaging device according to claim 1further comprising a composition unit that composes a high dynamic rangeimage by using the first signal and the second signal selected as theimage forming signal of the plurality of pixels.
 13. A signal processingdevice configured to process a signal output from an imaging elementthat includes a plurality of pixels each having a photoelectricconverter and outputs, from each of the plurality of pixels, a firstsignal based on charges generated by the photoelectric converter duringa first exposure time and a second signal based on charges generated bythe photoelectric converter during a second exposure time that isshorter than the first exposure time, the signal processing devicecomprising: a decision unit that decides whether or not a predeterminedvalue is exceeded for the first signal output from each of the pluralityof pixels and outputs a decision signal indicating a decision result;and a signal select unit that selects one of the first signal and thesecond signal as an image forming signal of each of the plurality ofpixels, wherein based on the decision signal of a target pixel and thedecision signal of another pixel arranged in a predetermined regionincluding the target pixel, the signal select unit selects the imageforming signal of the target pixel.
 14. An imaging system comprising: animaging device comprising an imaging element that includes a pluralityof pixels each having a photoelectric converter and output, from each ofthe plurality of pixels, a first signal based on charges generated bythe photoelectric converter during a first exposure time and a secondsignal based on charges generated by the photoelectric converter duringa second exposure time that is shorter than the first exposure time; anda signal processing unit that processes a signal output from the imagingdevice, wherein the signal processing unit comprises a decision unitthat decides whether or not a predetermined value is exceeded for thefirst signal output from each of the plurality of pixels and outputs adecision signal indicating a decision result, and a signal select unitthat selects one of the first signal and the second signal as an imageforming signal of each of the plurality of pixels, wherein based on thedecision signal of a target pixel and the decision signal of anotherpixel arranged in a predetermined region including the target pixel, thesignal select unit selects the image forming signal of the target pixel.15. A movable object comprising: the imaging device according to claim1; a distance information acquisition unit that acquires distanceinformation on a distance to an object, from a parallax image based onsignals output from the pixels of the imaging device; and a control unitthat controls the movable object based on the distance information.